Add an optimization doc on TPU #21155

bvrockwell · 2025-07-18T00:52:25Z

Creating a place in the docs for TPU optimization tips, WIP

github-actions · 2025-07-18T00:52:32Z

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

🚀

gemini-code-assist

Code Review

This pull request adds a new documentation page with optimization tips for using vLLM on TPUs. The content is valuable and well-structured. I've identified a few issues, mainly related to broken or fragile links and a missing image, which could impact the user experience. I've also found a typo in a command-line argument that could cause issues for users who copy-paste it. Please see my detailed comments.

docs/configuration/tpu/README.md

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

bvrockwell · 2025-07-23T19:35:07Z

@Chenyaaang @yaochengji ptal

Chenyaaang · 2025-07-23T21:30:32Z

docs/configuration/tpu/README.md

+
+### **Optimize based on your data**
+
+#### *max model len vs. most model len*


why is this included in "reducing compilation time" section? because using max model len increases compilation time.

yaochengji

Thanks for the awesome documentation! Left a few comments!

yaochengji · 2025-07-23T20:56:38Z

docs/configuration/tpu/README.md

+
+#### Latency-throughput tradeoff
+
+As with rightsizing the number of chips for your workload, consider adjusting `--max-num-seqs` to fine-tune the latency-throughput balance. Decreasing `--max-num-seqs` in increments of 128 and/or increasing the number of chips can help reduce latency.


I don't think max-num-seqs needs to be decreased by 128 each time.

yaochengji · 2025-07-23T20:57:50Z

docs/configuration/tpu/README.md

+
+This approach serves as a general rule of thumb. 
+
+#### Latency-throughput tradeoff


Another tradeoff consideration is max-num-batch-tokens, e.g. usually max-num-batch-tokens=512 has better latency but poorer throughput compared with max-num-batch-tokens=1024

yaochengji · 2025-07-23T21:40:21Z

docs/configuration/tpu/README.md

+
+Although the first compilation can take some time, for all subsequent server launches, vLLM can load these graphs directly from the cache, eliminating the compilation time for future runs. 
+
+Use `VLLM_XLA_CACHE_PATH` environment variable to write to shareable storage for future launches. 


Do we need to mention that there's actually a issue in persistent compilation cache?

yaochengji · 2025-07-23T21:44:38Z

docs/configuration/tpu/README.md

+
+If most of your requests are shorter than the maximum model length but you still need to accommodate occasional longer requests, setting a high maximum model length can negatively impact performance. In these cases, you can try introducing most model len by specifying the `VLLM_TPU_MOST_MODEL_LEN` environment variable.
+
+For example, 1% requests are 32k length and 99% requests are 2k length. You can pass 32k into `--max-model-len 32000` and use `VLLM_TPU_MOST_MODEL_LEN=2000`.


In this example, we'd better change max-model-len to 32768, and VLLM_TPU_MOST_MODEL_LEN to 2048.

hmellor

Thanks for the docs contribution!

I have some general style comments initially:

Please fix the pre-commit and DCO checks
Please avoid using bold text in the headings
Some of the headings are quite long might not render how it was intended
In the navigation bar this page is nested in a TPU directory and you only see the title of the page if you try and look inside it

hmellor · 2025-07-24T11:01:13Z

docs/configuration/tpu/README.md

Why docs/configuration/tpu/README.md instead of docs/configuration/tpu.md

hmellor · 2025-07-24T11:04:21Z

docs/configuration/tpu/README.md

+#### **SPMD**
+More details to come.
+
+#### Want us to cover something that isn't listed here? Open up an issue please and cite this doc. We'd love to hear your questions or tips. 


This should't be a heading, the whole thing appears in the ToC

hmellor · 2025-07-24T11:05:28Z

docs/configuration/tpu/README.md

+
+### **Get started**
+
+Looking for setup and installation instructions? Find them [here](https://docs.vllm.ai/en/latest/getting_started/installation/google_tpu.html).


Please use relative md links for internal docs references (http links will break in future releaes)

hmellor · 2025-07-24T11:05:37Z

docs/configuration/tpu/README.md

+
+#### **Profiling**
+
+The auto-tuner provides a profile of optimized configurations as its final step. However, interpreting this profile can be challenging for new users. We plan to expand this section in the future with more detailed guidance. In the meantime, you can learn how to collect a TPU profile using vLLM's native profiling tools [here](https://docs.vllm.ai/en/latest/examples/offline_inference/profiling_tpu.html). This profile can provide valuable insights into your workload's performance.


Please use relative md links for internal docs references (http links will break in future releaes)

hmellor · 2025-07-24T11:07:01Z

docs/configuration/tpu/README.md

+### **If possible, use the precision that matches the chip’s hardware acceleration**
+
+- v5e has int4/int8 hardware acceleration in the MXU
+- v6e has int4/int8 hardware acceleration in the MXU
+
+### **Don't set TP to be less than the number of chips on a single-host deployment**


These headings are also quite long

bvrockwell and others added 9 commits July 15, 2025 14:29

draft

336ce6f

rename

ecd6308

add tuner

de7986e

add information about calculator

19164c8

updating the docs

a0eedfa

Merge branch 'vllm-project:main' into main

0585f3b

update

62e520e

add link to more docs

caf6377

Merge branch 'vllm-project:main' into main

9d4a924

bvrockwell requested a review from hmellor as a code owner July 18, 2025 00:52

mergify bot added documentation Improvements or additions to documentation tpu Related to Google TPUs labels Jul 18, 2025

gemini-code-assist bot reviewed Jul 18, 2025

View reviewed changes

bvrockwell marked this pull request as draft July 18, 2025 00:53

bvrockwell and others added 11 commits July 17, 2025 17:56

Update docs/configuration/tpu/README.md

a764418

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

expand on certain topics

415de9c

clean up

95c0174

Merge branch 'vllm-project:main' into main

bcd39b2

Update docs/configuration/tpu/README.md

ae71890

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

update

d6d6127

update

e24df03

update calculator URL

cd36e2a

update image path

e3f9818

update

8527af4

fix image

661debe

bvrockwell marked this pull request as ready for review July 23, 2025 19:34

Merge branch 'vllm-project:main' into main

5421963

Chenyaaang reviewed Jul 23, 2025

View reviewed changes

yaochengji reviewed Jul 23, 2025

View reviewed changes

updates

6f581e8

hmellor requested changes Jul 24, 2025

View reviewed changes


		### Optimize based on your data

		#### max model len vs. most model len


		#### Latency-throughput tradeoff

		As with rightsizing the number of chips for your workload, consider adjusting `--max-num-seqs` to fine-tune the latency-throughput balance. Decreasing `--max-num-seqs` in increments of 128 and/or increasing the number of chips can help reduce latency.


		This approach serves as a general rule of thumb.

		#### Latency-throughput tradeoff


		Although the first compilation can take some time, for all subsequent server launches, vLLM can load these graphs directly from the cache, eliminating the compilation time for future runs.

		Use `VLLM_XLA_CACHE_PATH` environment variable to write to shareable storage for future launches.


		If most of your requests are shorter than the maximum model length but you still need to accommodate occasional longer requests, setting a high maximum model length can negatively impact performance. In these cases, you can try introducing most model len by specifying the `VLLM_TPU_MOST_MODEL_LEN` environment variable.

		For example, 1% requests are 32k length and 99% requests are 2k length. You can pass 32k into `--max-model-len 32000` and use `VLLM_TPU_MOST_MODEL_LEN=2000`.


		### Get started

		Looking for setup and installation instructions? Find them [here](https://docs.vllm.ai/en/latest/getting_started/installation/google_tpu.html).


		#### Profiling

		The auto-tuner provides a profile of optimized configurations as its final step. However, interpreting this profile can be challenging for new users. We plan to expand this section in the future with more detailed guidance. In the meantime, you can learn how to collect a TPU profile using vLLM's native profiling tools [here](https://docs.vllm.ai/en/latest/examples/offline_inference/profiling_tpu.html). This profile can provide valuable insights into your workload's performance.

Uh oh!

Add an optimization doc on TPU #21155

Are you sure you want to change the base?

Add an optimization doc on TPU #21155

Conversation

bvrockwell commented Jul 18, 2025 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Jul 18, 2025

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

bvrockwell commented Jul 23, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

yaochengji left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

hmellor left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

bvrockwell commented Jul 18, 2025 •

edited by github-actions bot

Loading

hmellor left a comment •

edited

Loading